DataFrame Select Operation

Selecting columns, projecting values, performing on-the-fly math calculations, and applying SQL expressions in PySpark.

What is the Select Operation?

The select() operation projects a set of expressions or columns, similar to the SELECT clause in relational SQL. In PySpark, it can take raw column names as strings, Column objects, or SQL expressions using selectExpr().

It is a lazy transformation that returns a new DataFrame with only the selected/calculated columns.

Syntax and Column References

You can reference columns in PySpark in three distinct ways:

from pyspark.sql.functions import col

# Method A: String representation (Simpler, but no column transformations allowed)
df.select("name", "salary")

# Method B: Column Object reference (Allows arithmetic and conditional operations)
df.select(col("name"), col("salary") + 5000)

# Method C: SQL Expression string projection (Using selectExpr)
df.selectExpr("name", "salary + 5000 AS new_salary", "upper(department)")

Example Usage Pipeline

Below is a complete, copy-paste-ready PySpark script demonstrating advanced selection techniques:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, upper, concat_ws

# 1. Setup local Spark session
spark = SparkSession.builder \
    .appName("DataFrame Select Demo") \
    .master("local[*]") \
    .getOrCreate()

# 2. Dummy dataset (Employees)
data = [
    ("usr_001", "John", "Smith", 90000, "Engineering"),
    ("usr_002", "Jane", "Doe", 110000, "Marketing"),
    ("usr_003", "Robert", "Johnson", 75000, "Engineering"),
]
columns = ["emp_id", "first_name", "last_name", "salary", "department"]
df = spark.createDataFrame(data, columns)

# 3. Apply Select Transformations
# - Combine first and last names into a single column
# - Calculate taxed salary
# - Convert department name to uppercase
selected_df = df.select(
    col("emp_id"),
    concat_ws(" ", col("first_name"), col("last_name")).alias("full_name"),
    (col("salary") * 0.85).alias("net_salary"),
    upper(col("department")).alias("dept_upper")
)

# 4. Apply selectExpr for SQL-style on-the-fly calculations
expr_df = df.selectExpr(
    "emp_id",
    "first_name || ' ' || last_name AS full_name",
    "salary * 0.85 AS net_salary",
    "upper(department) AS dept_upper"
)

# 5. Show results
print("=== Output from DSL select() ===")
selected_df.show(truncate=False)

print("=== Output from selectExpr() ===")
expr_df.show(truncate=False)

Rendered Output:

=== Output from DSL select() ===
+-------+--------------+----------+-----------+
|emp_id |full_name     |net_salary|dept_upper |
+-------+--------------+----------+-----------+
|usr_001|John Smith    |76500.0   |ENGINEERING|
|usr_002|Jane Doe      |93500.0   |MARKETING  |
|usr_003|Robert Johnson|63750.0   |ENGINEERING|
+-------+--------------+----------+-----------+

=== Output from selectExpr() ===
+-------+--------------+----------+-----------+
|emp_id |full_name     |net_salary|dept_upper |
+-------+--------------+----------+-----------+
|usr_001|John Smith    |76500.0   |ENGINEERING|
|usr_002|Jane Doe      |93500.0   |MARKETING  |
|usr_003|Robert Johnson|63750.0   |ENGINEERING|
+-------+--------------+----------+-----------+